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The Graduate Record Examinations (GRE) program is described in this article 
from the standpoint of the policy-making and administrative arrangement which 
provide the context within which the Advanced Literature and 22 other similar tests 
exist. The development and dissemination of the Advanced Tests is outlinecl A 
discussion of policy-making and control covers the functions of the recently founded 
GRE Board and its relationship to the Educational Testing Service. Considerable 
attention is devoted to such achievements of the GRE Board as (1) the separaflon of 
the Institutional Testing Program from the National Program for Graduate 
Selection. (2) improving communication between graduate education and the GRE 
program. (3) providing better interpretive aids for test scores and evidences of test 
validity, and (4) an interest in research. (AT) 
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THE GRADUATE RECORD EXAMINATION PROGRAM 

By John A« Winterbottom, Senior Program Director, 
Graduate and Professional Programs, ETS 
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Professor Booth’s paper describes in some detail the Advanced Literature in English 
Test of the Graduate Record Examinations and the activities and , 

examination committee which is responsible for producing the Test. It 
tell you something about the GRE program as a whole-that 

and aLinistrative arrangements which provide the context within which the Advance 
Literature Test exists and which it shares with twenty-two other tests. 
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I hope that I will be forgiven by those of you who already know che program well rf I 
begin by describing briefly, at a rather basic level, whac program rs and does. 

It has been ray experience that, in discussing the program, it is the better part o 
wisdom to asZne virtually total ignorance. I will come shortly to mtters in which 
I think even the most knowledgeable of you will find a modicum of interest. 
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It is the purpose of the program to make available a series of tests which can be 
used in admitting students into graduate study, primarily at the doctoral Uvel, and 
in granting fellowship awards and financial assistance. In 

program provides a series of twenty-three tests, om of them an Aptitude Test fera 
*iL separate Verbal and Quantitative scores are derived, the remainder ^ 

so-called Advanced Tests which are achievement tests in the areas of the most commonly 

taught undergraduate majors. 



The tests are given on six occasions during the year at centers established by the 
Educational Testing Service throu^out the world. Prospective graduate sttrients who 
are required or advised to take one or more of the tests by the schMls to which th y 
are applying register directly with ETS. They are sent tickets admitting them to 
centers whifh tLy choose from a published list; they take tte '"f 
and proctors employed by ETS; their answer sheets are scored centrally 
month after the administration of the test, score reports are sent to graduate 
designated by the candidates. Scores are kept on file for later reporting to additional 
schools if the candidate so directs. The system is designed in such a way *at rte 
graduate school or department need only announce its requirement or recommendatim that 
specified GRE tests be taken. Arrangements to take the tests tten become e J 
of negotiation between ETS and the candidate. In due course, if all goes well, the 
scores will appear on the desk of the appropriate official in the paduate school and 
they can then be made part of the record on which decisions regarding admission and 

financial support are based. 



The GRE program has been with the Educational Testing Service during the twenty yeps 
of the liter’s existence and, as a matter of fact, its origin antedates the founding 
of ETS by some fourteen years. During that time, the program has grown eprmously, 
partly as the result of an increasing tempo of adoption by graduate schools pd par y, 
So doSbt, as a concomitant of the greatly increased number of students entering graduate 
education. During the past academic year, over 190,000 candidates took one or more 
the tests. The corresponding number will exceed 200,000 during the current academic 
year. Last year, some 11,700 students took the Advanced Literature Test. 



Ill 

V- 



m 



o 

5 



s ^ 
* 



What I have been describing so far are the processes by which the tests are brought 
to the candidates and by which the scores reach the schools. Equally important, and 
perhaps of greater interest to this audience, is the process whereby the Advanced Tests 
are developed, more particularly the process by which the Advanced Literature Test 
comes into being. Professor Booth’s paper provides an admirably complete description 

Association of Departments of English, 1969 
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of the Test as it now exists, of considerations which shaped its design, and of some of 
the Committee *s concerns as it looks to the future, let me supplement his remarks, if 
I may, with a brief account of the steps leading to the production of a form of the 
Advanced Literature Test, 

ETS has in its Test Developme.it Division a large number of people who combine expertise 
in the techniques of testing with competence in a particular subject. While this 
Division can produce and does produce many tests from its own resources, it does not have 
within its own ranks the number of people nor the variegation of background that would 
enable it to produce unaided the amount and variety of test material demanded by all 
the tests for which it is responsible. It is customary, then, in a program such as the 
GRE or the College Board where large numbers of achievement tests are required--partic- 
ularly when the test must reflect changing conditions in the educational field they 
represent— to call on the services of examination committees recruited from appropriate 
fields and educational levels. 

In the GRE program, such committees are selected with the approval of the appropriate 
professional association. In the case of the Advanced Literature Test, the Executive 
Secretary of the MIA is asked to give his estimate of the probable suitability of 
various named individuals for work on the Examination Committee. The work of the 
Committee and the considerations which go into its selection are described to provide 
criteria in terms of which he can make his evaluations. He is also asked to provide 
additional names of possible committee members. All of this is done informally and 
in no way implies official endorsement or sponsorship of the Committee or of the Test 
it produces by the Association. The final selection of the committee members is made 
by ETS from a panel of names for which the approval of the appropriate professional 
association has been obtained in the sense and in the manner described. 

The composition of the committee is determined by a number of factors: the necessity 
of giving representation to various subspecialities within general fields; the desir- 
ability of having some geographic spread; the need to represent various approaches to 
the teaching of the subject; the likelihood that a prospective member of the committee 
will be comfortable with objective techniques and that he will become adept at writing 
multiple-choice questions (this is matter on which judgments are often difficult to 
make until after the first term of appointment) ; our luck in persuading busy people 
to take on yet another responsibility at a less than princely emolument. Appointment 
is characteristically for two years. 

Once appointed, the committee becomes responsible for a number of crucial functions 
in the production of a test. First, it must create a design for the test, if the 
test is one which is just being introduced into the program. Alternately, if a 
previous committee has worked on the test, the test design which governed its work 
is reviewed by the new committee and revised where necessary to make it reflect more 
accurately the current state of education in the subject concerned or to embody the 
ideas of the new committee about what information is needed to make reasonable admissions 
decisions at the graduate level. Second, the committee is responsible for writing a 
large share of the questions which will eventually be included in the test. In some 
instances— and this is more characteristic of tests in the sciences— additional question 
writers are hired to strengthen a test in a field in which no committee member happens 
to have specialized. In any case, whether all questions are written by the committee 
or whether supplemental writing is needed, all questions considered for the test are 
reviewed by all members of the committee. On the basis of their criticisms, questions 
are discarded, revised by the committee, or retained in their original form— this last 
being somewhat of a rarity. Finally, the committee decides which questions will be 
iiicluded in a particular form of the test. In doing so, it endeavors to make the test 
conform as closely as possible to the test design on which it earlier agreed. 
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In all these activities, the committee, working within the general framework of policy 
as determined by the GRE Board (about which more in a moment), has final responsibility 
for the design and content of the test. It is quite obvious, I think from Professor 
Booth* s paper that no member of the Committee for the Advanced Literature Test is 
wholly satisfied with it. It is an evolving instrument — it has undergone marked changes 
during its career thus far and the Committee contemplates further change. It is im- 
proving and is therefore, ipso facto , imperfect. The Committee and the ETS staff 
members who work on the Test are sustained in their efforts by a belief that, even in 
its present form, the Test can be of genuine though limited use to those charged with 
selecting students for admission to graduate departments of English. Beyond that be- 
lief is the sense that change is being encouraged in the Program and the reasonable 
hope that, even within the framework of multiple-choice techniques, yet more relevant 
measurement is possible. 

One word, before leaving this discussion of the Committee's work, about the role of 
ETS in producing the Advanced Literature Test. At the simplest level, it provides the 
staff support which enables the Committee to carry on its work effectively. In addition, 
it functions as the executive arm of the GRE Board and in that capacity has the respon- 
sibility for communicating to the Committee the policy decisions of the Board which are 
likely to affect its work. As I indicated earlier, the personnel who deal directly 
with the test committees have two kinds of professional competence: they are knowl- 

edgeable about the psychometric considerations which must be kept in mind in the pre- 
paration of satisfactory tests and they are familiar with the field in which the test 
is being assembled. They are thus in a position to advise the committee on technical 
matters, particularly as these involve the results of various types of statistical 
analysis which are carried out during the preparation of a test, and at the same time 
they are able to participate in discussions of the design and content of the test as it 
reflects the academic field. On occasion, they may write questions for consideration 
by the committee. 

Having provided some notion of how the Advanced Tests are produced and how they are 
made available to candidates, I should like now to comment on certain aspects of the 
control of the program and on the process by which program policy is formulated. 

Perhaps the single most important event in the recent history of the program was the 
founding of the GRE Board. Prior to January 1966, the program was under ETS ownership 
and control. The views of the graduate school community as to how the program should 
operate and its advice regarding the handling of particular problems came to ETS through 
the Committee on Testing of the Association of Graduate Schools, which met at least 
once annxially with ETS staff members who were responsible for the progrm. In 1964, a 
group of deans from schools located on the West Coast became interested in the GRE pro- 
gram and, more particularly, in how to provide for more direct and continuing control 
by graduate schools. They asked the Council of Graduate Schools in the United States 
to consult ETS about how this might be accomplished. A communication in this vein from 
CGS to ETS resulted in negotiations involving those two organizations and AGS, with the 
result that, in January 1966, the GRE Board was set up under the joint sponsorship of 
the three bodies. 

The Board consists of 16 members, most of them deans of graduate schools, four of whom 
are appointed by CGS and four by AGS; the remaining eight are elected by the Board 
itself. Members serve four-year terms. 

Under the terms of the original agreement, the Board determined policy governing test 
development and research in the program. During the past year, that power was broadened 
to cover all aspects of the program. Income from test fees is at the disposal of the 
Board and is used, first, to support continuing operations associated with providing 
the necessary tests, administering them, and reporting the scores. Any surplus of 






Income over expenses resulting from those activities reverts to a reserve fund which 
is available to the Board for research or for other activities it may choose to support. 

I should note here that, during the past year, negotiations were completed whereby the 
Graduate School Foreign Testing Program was turned over to control of the Board. This 
is a program which provides tests in the four languages, French, German, Russian, and 
Spanish, for use in determining whether a student has met the Ph.D. language require- 
ment. Since this program is intended to serve much the same constituency as the GRE 
program, it seemed appropriate to link them at the policy-making level. 

The role of ETS relative to the GRE Board is three- fold. First, it provides staff 
support for the Board* s activities in that it makes arrangements for meetings of the 
Board and its committees, manages the Board’s fiscal affairs, provides position papers 
and background papers on matters coming before the Board for discussion, prepares pro- 
ject proposals at the Board’s direction, or may on occasion present proposals on its 
own initiative. Second, it provides technical advice on matters having to do with 
test development, research design, and other matters of a professional nature in the 
general field of testing. Third, it acts as the executive arm of the Board in that it 
carries out or arranges to have carried out such projects as the Board decides to embark 
on. 

How do the Board and ETS feel about each other after nearly three years of living 
together? There is reason to think that the Board started with some initial skepticism 
regarding the reality of its control over the program. There was perhaps a lingering 
suspicion that the Board was intended as a figurehead and that the program would under- 
go little change as the resulty of anything the Board could do. This has not turned 
out to be the case. In its relatively brief history the Board has already made its 
impact felt in far-reaching ways. In one or two instances, it has brought about changes 
which ETS would probably not have undertaken on its own and about which it had some 
initial reluctance. The Board is now quite satisfied, I think, that its influence over 
the GRE program is paramount — that it is, indeed free to shape' the program as it will. 

ETS, for its part, was, as I have hinted, somewhat unprepared for the radical nature 
of some of the Board’s actions. It has, however, after some initial protest, accepted 
such actions with good grace and has clearly communicated to the Board its understanding 
that the Board has ultimate responsibility for shaping the destiny of the program. It 
has been greatly impressed by the caliber of men who have become members of the Board— 
they include the president of CGS, a former chairman of CGS, the president of AGS, and 
the former secretary of AGS. Even more giatifying has been the willingness of busy 
men such as these to become deeply involved in the concerns of the program, to spend 
time in the meetings of the Board and its committees, and to think seriously about the 
problems of the program. 

In the balance of my paper, I should like to review for you some of the major accomplish- 
ments of the GRE Board, partly because of their intrinsic interest, partly because they 
indicate the extent to which a new wind is blowing in the program. 

One of the early decisions of the Board was that the so-called National Program for 
Graduate School Selection should be separated from the Institutional Testing Program of 
the GRE. These two programs had existed in a symbiotic relationship, making use of the 
same set of tests but directed toward different ends and utilizing different adminis- 
trative arrangements. The National Program was intended to provide information which 
would assist in making admissions decisions at the graduate level. Through it the tests 
were made available on fixed dates and at centers established by ETS throughout the 
world. Candidates registered directly with ETS and scores were reported to graduate 
schools as directed by the candidates. The Institutional Testing Program was intended 
for use of undergraduate institutions wishing to test their senior students for purposes 
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of institutional self-evaluation or as part of the examination structure for determining 
student competence. The tests were given by the institution, on dates of its own choos- 
ing, and under its own administrative arrangements. The Board felt that tests construct- 
ed for the rather disparate purposes of the two programs could not be ideally suited to 
graduate admissions. It therefore decided that the two programs should be spilt apart, 
leaving each free to go its own way in pursuit of its own ends. The examination commit- 
tees for the GRE program have been told that henceforth the Advanced Tests are to be 
constructed for the sole purpose of serving the graduate admissions function. Another 
set of tests offered through another program will be offered for the use of undergrad- 
uate institutions beginning in the fall of 1969. 

What I have described is more than a mere organizational shuffle. It represents a real 
shift in the philosophy governing the GRE program. It means that the emphasis on com- 
mand of the content of undergraduate courses, which has hitherto characterized the 
tests, need not be maintained if, in the judgement of the examination committee, the 
graduate admissions function could be better served by another kind of test. This is 
not to say that attention to achievement at the undergraduate level will forthwith dis- 
appear. It is quite possible that a given examination committee, after taking soundings 
in the relevant academic constituency, will decide to maintain this emphasis in its 
test. But the possibility of change has been opened up and, to the extent that there 
are factors other than undergraduate achievement which are relevant to graduate per- 
formance and to the extent that means of measuring these in a mass testing program can 
be devised, these factors can be reflected in the tests. 



Another major contribution of the Board has been the facilitating of communication be- 
tween graduate education and the GRE program. Such communication has, of course, been 
vastly strengthened by the mere existence of the Board itself. Through it the program 
has achieved contact with more than 20 major graduate institutions and with organizations 
such as AGS, CGS, and the National Research Council. This has had the dual effect of 
bringing the program to the attention of potential users of its services and of open- 
ing up resources which can be called upon to assist in various projects intended for 
the improvement of the program. 

Under stimulus from the GRE Board, efforts are being made to foster in the academic 
community a better understanding of what the tests are like, what they are Intended to 
do and, equally important, what they make no pretense of doing. As Professor Booth's 
paper seeks to make clear, the Advanced Literature Test attempts to measure not only 
command of the factual background of literature in English but also skill in literary 
analysis and sensitivity to literary values, and it is probably more successful in this 
endeavor than most people having only a casual acquaintance with the test give it credit 
for. On the other hand, a multiple-choice test is incapable of measuring directly the 
power of self-expression or the ability to produce and organize ideas of one's own. It 
is limited to the kinds of measurement which can be effected through requiring the can- 
didate to recognize what is true or correct when it is presented to him. As those most 
closely associated with the GRE program realize— and here I include the ETS staff, the 
various examination committees, and members of the GRE Board — the test scores do not 
in themselves provide an adequate basis for making admissions decisions. What they can 
do, even in their present form, is supplement information derived from undergraduate 
transcripts, application forms, and the opinions of undergraduate instructors and en- 
hance the soundness of judgments based on these traditional criteria of academic promise. 

But, though those who work on the tests are convinced that they are useful in their 
present form, they are equally aware that improvement is possible, even necessary. And 
it is our hope in establishing closer contact vrith the academic community not only to 
promote understanding of the tests as they are, but to enlist help in making the tests 
better. To this end, the Board has decided to embark on a program of creating review 
panels for specified Advanced Tests whose task it will be to consider broadly the 
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qualities sought by graduate departments in their admitted students, to review admissions 
practices prevalent in the field, to define an appropriate role for the test scores, 
and to make recommendations regarding possible revisions in the design of the tests. 

The panels will be as broadly representative of the field as they can be made without 
becoming unwieldy and they will include representation from the appropriate examination 
committee. They will be jointly appointed by the GRE Board and the appropriate profes- 
sional association and so that they will function not only under the auspices of the 
GRE program but with the interest and involvement of the profession. Five such panels 
will be appointed this academic year and, if these are successful, additional panels 
will be appointed next year. No commitment has been made with respect to a review panel 
for the Literature Test, but the likelihood is that the MIA will be asked to cooperate 
in the formation of such a panel next year. 

Another area in which the GRE Board has been active is in the provision of better inter- 
pretive aids for the test scores. Considered in isolation the scores are virtually 
meaningless. They take on significance only as means are provided for using them in 
making comparisons among the individuals taking the test. One way of doing this is to 
provide norms tables showing the percentage of a defined group of candidates scoring 
below various points on the score scale. Prior to this year, the comparision group con- 
sisted of seniors of undergraduate colleges who took the tests in the early 1950* s. A 
problem was that the ability level of these relatively unselected seniors was lower than 
that for the more select group who characteristically apply to graduate schools. For 
this reason, the norms tables provided very little discrimination among applicants to 
some graduate departments, all of whom tended to have scores at the upper end of the 
score range. The Board decided that, beginning last year, norms would be based on the 
candidates taking the tests over the previous three years. As a result, although a 
given test score continues to represent the same level of competence, it will correspond 
to a different percentage point in the norms table. For example, a score of 700 on the 
Advanced Literature Test will continue to represent about the same level of preparation 
as has been true in the past. However, it will correspond to a percent of 89 in the 
current norms table, as opposed to a percent of 93 on the 1952 norms. Our belief is 
that comparisons made possible by the current norms based on recent candidate groups 
will provide a more realistic basis for evaluating scores than did the older norms. 

The Board has also expressed its concern about providing evidence of the validity of 
the tests as predictors of graduate success. A series of studies has just been reported 
which indicate that, although none of the quantifiable information available at the 
point of admission does a particularly good job of predicting success, the test scores 
perform creditably in comparison with the undergraduate record. There is good reason 
to believe that the failure of any of the predictors used in these studies to display 
substantial validity as "measured by statistical means is attributable to the great 
difficulty of obtaining satisfactory measures of student performance at the graduate 
level. This round of validity studies included students in only six subjects and we 
hope to extend our efforts into additional fields. 'We hope also to be able to effectuate 
methodological improvements based on our recent experience which will enhance the signif- 
icance of the results we obtain in future studies. 

Finally, in this rehearsal of major accomplishments of the GRE Board, let me mention 
its interest in research. As a matter of long range investigation, the Board has iden- 
tified as one of its major concerns the development of some means to measure potential 
productivity and originality. One study in this area is already in progress. It involves 
an attempt to devise a method of taking an inventory of a student's previous accomplish- 
ments which were to a large extent self- initiated and were carried on outside the re- 
quirements of his academic experience, which displayed some measure of originality, and 
for which some degree of public recognition has been achieved. The attributes which we 
are attempting to observe and study are, of course, highly resistant to objective 
description and measurement. They are, however, of overriding importance in the estima- 
tion of those responsible for the admission and instruction of graduate students and the 
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Board' feels obligated to take the first steps along a path which will perhaps lead to 
a better understanding of these qualities. 

I think it is evident from what I have said that, brief though its history has been, 
the interests of Che Board have undergone a remarkable process of expansion and prolif- 
eration. In many instances, the projects it has set in motion have been clear-cut as 
to purpose and method-- for example, the provision of three-year rolling norms. In two 
areas of interest, however, it is venturing into territories for which no charts exist. 

I am thinking of its concern with revising the Advanced Tests--a process which must be 
preceded by a prolonged effort to find out what is needed and wanted by the graduate 
schools— and of its desire to investigate in pioneer fashion the possibility of measuring 
attributes which are not now touched on by the Graduate Record Examinations nor by other 
instruments suitable for operational use in making critical decisions. In order to pro- 
vide direction and continuity to its efforts along these lines, the Board decided at its 
last meeting to establish a Test Development and Research Committee. It will be the 
general responsibility of the Committee to effectuate the policies of the Board as these 
have Implications for test development and research. Exactly how it will set about do- 
ing this remains to be seen since the Committee has yet to meet for the first time. How- 
ever, it is very probable that it will find itself trying to make some systematic in- 
vestigation of the needs of graduate education in the matter of admissions, to determine 
how the tests should be reshaped to meet these needs, and to recommend to the Board 
specific projects of a developmental or exploratory nature. At any rate, it is a 
certainty that the new Committee will become a major vehicle through which the forces 
for change which are at work in the program will find expression. 

In concluding this paper, let me briefly summarize what I conceive to be the situation 
of the GRE program at this point in time. Looking ahead, one can foresee the strong 
possibility that the Graduate Record Examinations will play an increasingly important 
role in graduate admissions. For one thing, the pressure of numbers is likely to be- 
come greater and graduate departments v/ill be seeking help in making admissions wherever 
they can find it. Again, some of the traditional admissions criteria are likely to be- 
come less useful. As the nuinber of applicants to a given graduate department multiplies, 
the range of their undergraduate origins will expand, and letters of recommendation 
from Instructors familiar to the admitting department will not always be available. 
Furthermore, if present portents have any validity, the undergraduate record will be- 
come a less useable criterion as more and more undergraduate institutions resort to 
reporting only in pass-fail categories. As numbers of applicants increase and as 
discriminations among them become more difficult, one consequence is likely to be in- 
creased reliance on the test scores. 

The Graduate Record Examinations are made available through a mass testing program— in 
excess of 200,000 candidates are tested per year and scores go to more than 600 institu- 
tions. This situation brings with it its administrative exigencies. There are certain 
examination techniques available to the teacher responsible for fifty students which 
are out of the question when numbers of candidates are measured in thousands. Never- 
theless, despite certain rigidities which are Inherent in the situation within which 
they function, the tests are capable, even as they stand, of providing useful information 
to those seeking to select students having a reasonable chance of success in graduate 
study. But beyond that, as I have tried to point out, the program is genuinely open to 
change. We are engaged in a serious effort to find out what the graduate departments 
demand of their admissions procedures and how the tests can be modified to play a more 
significant part in admissions decisions. We recognize that, as a means of stimulating 
change and of ensuring that changes will be meaningful, we must enlist the interest and 
assistance of graduate school faculties and administrations. My hope is that I have 
done some small service to that cause on this occasion. 
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